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ABSTRACT 



Minnesota's Family Service Collaboratives and Children's 
Initiative Partners encourage cooperation between communities and schools to 
improve the health of babies and children, increase school readiness, provide 
stable family environments, and promote basic academic skills. To fulfill the 
evaluation needs of these programs, this resource manual presents an 
evaluation workshop designed to assist collaborative teams in conceptualizing 
and specifying their evaluation designs using a common evaluation framework 
that will allow for both site-specific and cross-site reporting of outcomes. 
The materials included in this manual introduce a five-step process for 
specifying an outcome evaluation design: (1) defining and specifying the key 

elements of the collaborative initiative; (2) specifying evaluation claims 
about the observable effects of the initiative; (3) specifying the evidence 
or documentation that may be used to substantiate each claim; (4) creating c. 
map that links key elements and indicators; and (5) specifying an evaluation 
plan that includes level of evidence for each indicator, instruments and 
procedures, data analysis and standards of comparison. The manual discusses 
the role of evaluation consultants in assisting sites to specify their 
outcome education plans, and includes a form for sites to indicate the types 
of assistance that would be most helpful to them. Four background articles on 
potential outcomes and indicators, and a form for evaluating the workshop are 
appended. (JPB) 
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Introduction 



Purpose of the Evaluation Workshop 



The purpose of this evaluation workshop is to assist collaborative teams in conceptualizing and 
specifying their evaluation designs using a common evaluation framework that will allow for both 
site- specific and cross- site reporting of outcomes. Specifically, 



• Sites with local evaluators will be able to document their efforts using a 
common format. 



• Children's Initiative Partners will have the opportunity to plan an 
evaluation design for already specified outcomes and to assure their 
program elements link to these outcomes. 

• All other implementation sites will be given a process for 
conceptualizing and planning their local evaluations using a common 
format. 



• Members of the Children's Interagency Technical Assistance Team 
(ChITA) will have an opportunity to use the same evaluation framework 
to specify a plan for evaluating their efforts. 



A one-day workshop is only long enough to introduce collaborative teams to a common language 
and to share resources that will assist them in specifying their evaluation designs. We expect teams 
that have yet to formulate their outcome evaluation plans will need to continue their work once they 
return home. 

In part three of this manual, we describe how the evaluation consultants may be of assistance to 
sites as they work on the specification of their outcome evaluation plans. We also include a form 
for sites to indicate the types of assistance that would be most helpful to them. 



Evaluation Requirements 



The Family Services and Community-Based Collaboratives were initiated by Governor Arne H. 
Carlson and the Minnesota Legislature in 1993. Included in this initiative are grants to foster 
cooperation and collaboration and help communities come together to improve outcomes for 
Minnesota's children and families. By providing incentives for better coordination of services, 
Minnesota hopes to increase the number and percentage of babies and children who are healthy, 
children who come to school ready to learn, families able to provide a healthy and stable 
environment for their children, and children who excel in basic academic skills. 

There are really two major sources of funds to support the implementation of Family Services and 
Community-Based Collaboratives. Implementation grants are for communities that have developed 
measurable goals and a comprehensive plan to improve services for children and families. The 
grant funds must be used to provide direct services to children and families. 

Minnesota legislation for the Family Service and Community-Based Collaboratives includes the 
following language regarding evaluation of these initiatives: 

Collaboratives receiving implementation grants must submit a report to the Children's Cabinet. The report 
shall describe the progress the collaborative made toward implementing the local plan, how funds received 
were used, the number and type of clients served, and the types of services provided. The report shall be 
submitted to the Children's Cabinet by December 31, 1994. . .within two years of the date on which a 
collaborative receives an implementation grant, a collaborative shall submit a report to the Children's 
Cabinet describing the extent to which the collaborative achieved the outcomes developed under Minnesota 
Statutes section 121.8355. 

A second major source of funds for collaboratives is through the Minnesota's Children Initiative. 
Three collaborative sites have been selected to receive funds from the Pew Charitable Trust over a 
three year period. The Children's Initiative Partners are focusing their initial work on four pivotal 
outcomes for children ages 0 to 6 and their families: improved child health, adequate child 
development, reduced barriers to adequate school performance, and adequate family functioning 
and stability. 

Children’s Initiative partners are required to provide annual progress reports to the Pew Charitable 
Trusts (5/15/95, 5/15/96, and 8/15/97). Reports must include a thorough account of what was 
accomplished by the expenditure of funds, including a description of progress toward achieving 
the measurable objectives of the grant. 

In January of 1995, Minnesota Planning contracted with the University of Minnesota to assist in 
the design and implementation of an evaluation system for the collaborative initiatives. This 30- 
month effort is intended to achieve a number of related objectives: 

• To design a statewide evaluation system that builds on and augments the site-specific 
evaluation activities of each local collaborative; 

• To increase the capacity of collaborative grantees and partners to implement ongoing outcome 
evaluation activities; 

• To identify and communicate the characteristics, service components, and strategies for 
effective collaborative efforts to better serve the needs of families and children; 

• To identify and communicate common themes and strategies across effective collaboratives; 
and 

• To provide data that will inform recommendations for future collaborative initiatives. 
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Evaluation consultants from the University of Minnesota are following a three stage plan for 
addressing these objectives: 

Stage 1: Development of broad evaluation questions that will be answered through various 

data collection procedures and forms. A list of these guiding questions appears on 
the next page. 

Stage 2: Development/revision of reporting documents to include question items and 

response formats that provide information to answer the broad evaluation 
questions. Reporting documents include: 

A. the grant application form completed by collaborative sites; 

B . an agency report completed by state agencies involved with the collaboratives; 

C. a semi-annual progress report form (formerly the quarterly report form) 
completed by collaborative sites; 

D. an annual survey form completed by state agencies and collaborative sites; 

E. the final report submitted by each collaborative site to the Children's Cabinet 

Documents B-E are currently being revised and/or developed by the evaluation 
consultants in cooperation with the Children's Interagency Technical Assistance 
Team (ChITA) and Minnesota Planning. 

Stage 3: Development of comparable outcome evaluation designs across multiple sites. The 

materials included in this manual are intended to provide an overall 
framework for sites to use in specifying their outcomes and evaluation 
designs. 

Once these stages are completed, sites will be prepared to collect and report comparable outcome 
data (although the specification of outcomes and evaluation plans will vary from site to site). The 
role of the evaluation consultants from the University of Minnesota will then be to (a) provide 
technical assistance to the sites related to their evaluation designs; (b) summarize information from 
the reporting documents described above in order to answer the guiding evaluation questions; and 
(c) share findings related to the guiding evaluation questions with the sites, ChITA, the Evaluation 
Focus Team, and the Children's Cabinet for discussion. 



Key dates related to the development and implementation of the evaluation system include: 



EVALUATION TASKS AND TIMELINES 



COMPLETION 

DATE 


ACTIVITY 


RESPONSIBILITY 


2/17/95 


Overarching evaluation questions 
specified 


Bloomberg and 
Seppanen 


2/21/95 


Overarching evaluation questions 
disseminated to evaluation and local sites 
for review and comment 


Boston and 
Reardon 


2/24/95 


Deadline for comment on overarching 
evaluation questions 


Comments 
received by 
Bloomberg and 
Seppanen 


3/2/95 


Final draft of evaluation questions 
approved 


Children's 

Cabinet 


3/29/95 


Local evaluation plans developed 


All sites 


4/17/95 


ChlTA TEAM MEETING (present initial 
list of barriers, draft survey) 


Bloomberg , 
Seppanen 


5/19/95 


Local outcome evaluation designs 
submitted to evaluation consultants for 
compilation/clustering 


LOCAL SITES 


5/19/95 


EVALUATION TEAM MEETING 


— 


6/30/95 


Semi-annual progress report completed by 
local sites 


Local Sites 


6/30/95 


Annual Survey forms completed by 
identified stakeholders 


to be identified 


8/7/95 


ChlTA meeting (presentation of survey 
findings) 


Bloomberg and 
Seppanen 


2 years after 
receipt of 
implementation 
funds 


Outcome Reports completed by each local 
site; submitted to Children's Cabinet 


local sites 
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Broad Evaluation Questions 



The intent of this workshop is to prepare collaborative sites to specify and measure the outcomes of their efforts. 
The outcome data submitted by sites will be used to answer questions 6.1, 7.1-7.4 and 8.1-8.3. The evaluation 
consultants will summarize information from other sources (the grant applications, agency reports, semi-annual 
progress reports, and an annual survey) to answer the other broad evaluation questions listed here. 

CONTEXT 

1 . 1 What factors and community characteristics have influenced the design and implementation of the 
collaborative initiatives? 

1 .2 What issue(s) or problem(s) were the collaboratives designed to address? 

BARRIERS 

2.1 What are barriers to implementation of the collaborative initiatives at the state and local level? 

2.2 How have collaborative initiatives addressed local-level barriers? 

2.3 How have state-level barriers been addressed? 



COLLABORATIVE PROCESSES 

Involvement 

3.1 How are organizations/agencies, community groups, and families chosen to be directly involved in the 
collaborative initiative? 

3.2 Which organizations/agencies, community groups, and families are directly involved in the implementation 
of the collaborative initiative? 

3.3 What role(s) do participating organizations/agencies, community groups, and families play in the 
collaborative initiative? 



Governance 

4. 1 What governance structures are in place within each collaborative site? 

4.2 Who participates in the governance of the collaborative initiative? How are these participants chosen? 

4.3 What authority does the governing body have? 



Resources 

5. 1 How have grant funds been used? 

5.2 How much funding has been leveraged from other sources for use by the collaborative initiatives? 

5.3 To what extent are sites integrating funds and resources? 

Organization Elements 



6.1 * What are the key elements of the implementation plans for collaborative initiatives? 

6.2 To what extent are key elements culturally relevant? 

6.3 What progress have the collaborative sites made toward implementing the key elements of their local plans? 

OUTCOMES 
Systemic Change 

7.1* What types of systemic change do the collaborative initiatives expect to achieve? 

7.2 * What are the indicators of systemic change in collaborative sites? 

7.3 * What are the indicators of systemic change at the state level? 

7.4 * To what extent has systemic change occurred? 



Outcomes for Children, Youth, and Families 

8.1 * What types of outcomes have been specified by the collaborative initiatives? 

8.2 * What indicators substantiate the achievement of these outcomes? 

8.3 To what extent have the outcomes been achieved? 



*These questions will be specifically 



addressed at the evaluation 



workshop. 




10 : 



9 



Developing a Common 
Language 



Overview 



The purpose of this workshop is to assist collaborative teams in conceptualizing and specifying 
their outcome evaluation designs using a common format that will allow for both site-specific and 
cross-site reporting. This step is the summative or product phase of a program evaluation; it will 
yield data regarding the impact of the collaboratives on children, families, community, and the 
service system as a whole. 

Because the collaborative initiatives are in the early stages of implementation, it is important to 
understand how an evaluation of outcomes fits with other important evaluation approaches. These 
other approaches include: 

Needs assessment. A need is defined as the discrepancy between what is and what ought to 
be. Once identified, needs are typically placed in order of priority. They are the basis for setting 
goals and objectives. The planning grants awarded to collaboratives focused on the identification 
of needs and the articulation of collaborative goals. 

Program planning. From collaborative goals, specific, measurable objectives are specified and 
a plan containing the means to attain these objectives (procedures, strategies, and activities) is 
formulated. The planning grants allowed collaborative sites to begin program planning. Research 
on planning and implementation has demonstrated, however, that it is not a linear path from 
planning to implementation. Collaboratives will cycle back-and-forth between planning and 
implementation as they pilot, implement, revise, and ultimately institutionalize the key elements of 
their initiatives. 

Formative evaluation. There are two aspects to the formative or process phase of evaluation: 

• Progress evaluation activities focus on monitoring indicators of progress toward goals and 
objectives. The sites will report information regarding their progress as part of the semi-annual 
progress reports. 

• Implementation evaluation activities focus on the identification of discrepancies between the 
plan and reality at a given point in time. This activity keeps the initiative true to its design or 
modifies it appropriately. The sites will report information regarding the implementation of 
their plans as part of the semi-annual progress reports and an annual survey. Sites may also 
include an implementation evaluation as part of their local evaluation efforts. 

Beyond reporting on their progress and implementation status, collaborative sites may wish to 
implement evaluation procedures to allow local stakeholders to reflect on the effectiveness of a 
number of key dimensions of collaborative relationships. Sites may be particularly concerned with 
issues related to the quality of collaborative processes and the relationships among participant 
organizations and groups. The seven dimensions that are typically part of a process evaluation of a 
collaborative include the focus of the relationship (client, agency, or system), levels of staff 
involved, degree of formalization in terms of governance and decision making, resource 
involvement, focus of power, and degree of goal congruence. 

One aspect of the technical assistance available from the University of Minnesota consultants will 
focus on facilitating local collaborative sites in designing and carrying-out formative evaluation 
efforts. 



Outcome evaluation. This step represents the focus of the report due to the Children's Cabinet 
within two years of the date on which a collaborative receives an implementation grant. Although 
each initiative has its own special characteristics, we are striving toward creating a common 
language that will allow for communication and comparison among the different initiatives. The 
materials included in this manual introduce a five step process for specifying an outcome evaluation 
design: 

• Defining and specifying the key elements of the collaborative initiative; 

• Specifying evaluation claims about the observable effects of the initiative; 

• Specifying the evidence or documentation that may be used to substantiate each claim (these are 
called indicators ); 

• Creating a map that links the key elements and indicators; and 

• Specifying an evaluation plan that includes: 

- Level of evidence for each indicator 

- Sample 

- Instruments/procedures 

- Timing of measurements 

- Data analysis 

- Standard of comparison 

Each site will prepare and submit a statement of their outcome evaluation design (e.g., the key 
elements of their initiative, evaluation claims, indicators, a map that links the key elements and 
indicators, and evaluation plans). The evaluation consultants will provide feedback to the sites on 
their submissions and will prepare a summary report that describes the evaluation designs of all 
sites. This report will be shared with the collaborative sites, the ChITA group, and the Children's 
Cabinet. 
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Key Elements 



Definition of a Key Element 

Key elements refer to the strategies, activities, or mechanisms that will be implemented in a 
collaborative site to attain the goals and objectives of the initiative. 1 Key elements describe what 
the collaborative initiative looks like in practice as it is implemented. Most collaborative initiatives 
can be described with 10 to 15 key elements. The key elements of a collaborative initiative help to 
answer the following questions: 

1 . What are key players (organizations, staff, families, parents, children and youth) each doing as 
part of the collaborative that will result in the desired outcomes? What happens in a typical 
day/week/year? 

2 . What collaborative practices are central to achieving the outcomes and/or innovation? 

3. What would a visitor to your site see when the collaborative is fully operational? 



Instructions for Identifying Key Elements 

1 . Identify the types of key players who are part of the collaborative initiative (e.g., children, 
youth, parents, families as a whole, staff, organizations and groups). 

2. Brainstorm what each key player will do as part of the collaborative when it is fully 
implemented (refer to the categories of elements on the next page). 

3. Develop a sentence that describes the ideal use of each element The sentence describing the 
element should have a subject (the key player who implements or uses the element) and an 
action verb. The sentence should be as concrete as possible (e.g., The XYZ elementary school 
provides elementary-age children and their families with integrated services [specify what types 
of services] that are located at or directly linked to the school.) 

4 . (Optional) Since your collaborative initiative may involve a number of key elements that are 
not yet fully operational, identify other variations of each element that arc acceptable in the 
short term (e.g., The XTZ elementary school provides elementary-age children and their 
families with integrated services [specify a smaller set of services] that are directly linked to the 
school [but not yet located at the school]) 

Remember: Key elements are not the same as collaborative goals and objectives. 

Key elements are the strategies , activities, or mechanisms that will lead to the 

desired outcomes. 



1 The goals and objectives that have been specified by the collaboratives represent a mix of statements related to 
what the initiatives hope to accomplish (in toms of outcomes for children and youth, parents, families, and systemic 
change) and how they plan to accomplish these outcomes (in terms of key elements). When used here, the term 
"objective" refers to what the initiatives hope to accomplish in toms of outcomes. 
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Examples of Key Elements 



Most initiatives can be described with 10-15 key elements. Identify the key elements of your 
initiative that are critical to achieving the desired outcomes for children and youth, families, and the 
system as a whole. The following examples of key elements are from two sources: (1) a review 
of the collaborative and service integration literature, and (2) a review of the documents and reports 
that have been prepared by collaborative sites to date. Many of the phrases are general statements 
and with further refinement could serve as key elements. Use these phrases as a basis for 
discussion among your collaborative team. 

Organizations, staff, or community groups will . . . 

Outcomes Orientation 

• articulate a common set of desired outcomes for all children, youth, and families 
Collaborative Governance 

• implement a governance structure that includes families and service providers 

• create and maintain interagency agreements 

• establish policies regarding collaborative activities 

Comprehensive Planning 

• participate in a local collaborative council to coordinate services and provide overall direction 

• consult with school-age children and their families on the best way to serve them 

• identify the assets and needs of children and families in their communities and target resources 
to be redirected accordingly 

• host community meetings regarding resource allocation or service provision 
Information Management and Sharing 

• use computer networks to share client and agency information 

• provide a comprehensive community resource directory 

• provide a toll-free information and referral service 

Funding Integration and Reallocation 

• commit financial resources to an integrated fund 

• provide a direct-services flexible fund as a source of last resort resources for families and 
students 

• establish a wrap-around services fund 

• design services to meet the requirements for state/federal reimbursement or private grant funds 

• reinvest dollars saved into ongoing and broad-based local prevention efforts 
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Public Awareness 

• prepare and release information to the public 

• write and distribute a newsletter 

Case Management/Service Coordination 

• integrate assessment and referral activities to create a single referral and entry point for families 

• develop and carry out with each family a prevention-intervention plan 

• provide unitary case management for families needing referrals to multiple agencies 

• jointly review cases and minimize service duplication 

• have the authority to establish service eligibility of families 

Service Scheduling 

• expand the hours that services are available 

• provide staff during peak hours of service use 

• provide more drop-in times 

Service Accessibility 

• provide counseling and family services coordination for students and their families both in 
school and at home [this may actually involve multiple key elements] 

• conduct home-visits when transportation is a barrier to families accessing services 

• provide and coordinate services at an easily accessible location 

• provide information about transportation options 

• provide universal access to medical services at XYZ medical center 

Outreach 

• provide education and support during home visits to pregnant women and families with infants 

• provide periodic family visits to children identified as at-risk 

• provide a universal contact for pregnant women and parents with newborns 

• provide universal monitoring of child development from birth through age 4 

• provide a universal contact for early childhood screening and school registration 

• identify and contact children who may need crisis-oriented services 

• recruit parents and youth to participate in services and activities 

Service Provision 

• provide families a menu of service options to pick from 

• provide an after-school enrichment program, recreational activities, summer activities, etc. 

• provide expanded options for child care 

• provide a toy-lending library 

• provide adult basic education 
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• provide parenting education classes to [identify target group] 

• provide support groups 

• provide tutoring to [identify target group] 

• provide appropriate curricula on diverse cultures 

• provide in-home family therapy to [identify target group] 

• provide a day treatment program for mentally ill children [may actually involve multiple key 
elements] 

Service Integration 

• provide parenting classes and other services [specify] at a family resource center 

• provide child-care and other services [specify] at a center that is co-located with a business that 
provides jobs and job training opportunities 

• integrate all [specify service: housing, transportation, child care, etc.] services in the 
community 

• provide elementary age children and their families with school-based integrated services 
[specify types of services and location] for prevention and/or early intervention 

• provide jobs and job training at a family resource center 

Barrier Reduction 

• identify barriers to delivering services and strategies to eliminate barriers 

• eliminate barriers to integrating and reallocating resources for children and families 

• change rules, regulations, and mandates that interfere with the initiative’s work [or seek 
variances] 

StaffTraining 

• recruit and train volunteers 

• participate in cross-agency staff development activities 

• provide cross-training for staff across agencies/organizations in [specify training areas] 

Families will . . . 

• use transportation options 

• access services in a timely way 

• seek and choose services they feel are necessary to address their needs 

• participate in early childhood screening 

• participate in a comprehensive family assessment process 

• participate in ECFE 

• participate in home-based family therapy 
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• borrow toys from a toy lending library 

• participate in intensive services [specify types and how often] 

Parents will .... 

• participate in designing and developing a family center 

• receive prenatal care 

• attend workshops [specify types and how often] 

• provide a safe and secure environment for their children [should be more concrete] 

• seek information about support programs, services, and resources available 

• develop their own individualized prevention and remediation plan 

• solve their own problems 

• work with a case manager to identify needs and develop a family action plan 

• participate at community meetings regarding violence prevention 

• provide feedback to evaluate and improve the service delivery 

Children and youth will . . . 

• provide positive peer support 

• participate in health check-ups 

• participate in immunization clinics 

• participate in child care 

• attend school regularly 

• attend classes for teen parents 

• attend summer activities [specify types] 

• participate in special community projects [specify types and how often] 

• attend a day treatment program [includes two or three key elements] 
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Key Elements Worksheet 



Initiative Name: 



Statement of key element: 



Other acceptable variations: 



Statement of key element: 



Other acceptable variations: 



Statement of key element: 



Other acceptable variations: 



Statement of key element: 



Other acceptable variations: 
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Claims 



Definition of a claim 

A claim is defined as the observable effect of your initiative on children and youth, families, or on 
the service system as a whole. Claims are based on the measurable changes expected in the 
targeted participant group. 



Instructions for Identifying Claims 

1 . Succinctly state the major observable effects of your collaborative on the intended beneficiaries. 

2. Generally, a claim statement includes the target group and the nature of the change expected 

3 . Target groups may include children or youth, parents (or expectant parents), families, or the 
service system as a whole. 



4 . Changes may include 

A. Improved status in a particular domain (e.g. health, nutrition, physical development, 
language development); 

B . Changes in knowledge and skills; 

C. Improvement of attitudes and behaviors 

D. Improvement in institutional or system practices and procedures. 

5 . Changes in B, C, and D (above) should logically be expected to contribute to a change in A 
(above). 
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Examples of Claims 



• The number of cases of childhood diseases for which immunization is available will 
decrease in ABC county (child and family health). 



• Parental violence directed toward children will decrease (family functioning). 



• School-age students in ABC school will show improved academic achievement ( school 
performance). 



• Families will increase the time spent participating in intellectually challenging activities 
and using intellectually challenging materials (child development). 



• An increased proportion of parents whose children attend XYZ school will regulate the 
television viewing of their children (family functioning, child development). 



• By instituting an integrated fund, duplicate efforts [specify] will be eliminated and 
expanded opportunities [for what] will be provided at significantly reduced costs 
(systemic or institutional change). 
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Indicators 



Definition of an Indicator 

An indicator is the evidence or documentation you could use to substantiate a claim about the 
observable effect of your initiative. Think of indicators in terms of children and youth, family, and 
institutional outcomes that you could point to as evidence that the key elements of your 
collaborative have had an impact. 

In addition to thinking about long-term impact, teams should consider the identification of 
intermediate indicators that reflect the observable effects of the collaborative in the short-term 
(during the next two years). There are a number of reasons for also identifying intermediate 
indicators: 

• a number of intermediate variables may predict long-term changes in key outcome variables. 
For example, information about immunization rates can be collected in the short term and 
should predict longer-term changes in the incidence of certain communicable diseases; 

• collaborative initiatives vary in terms of their implementation status; it may not be fair to 
evaluate a new initiative during the next year or two in terms of long-terms effects; and 

• service use patterns, changes in attitudes, changes in behaviors, and changes in institutional 
practices represent an intermediate step in the implementation of a well-developed collaborative 
initiative and serve to indicate that progress is being made toward achieving the long-term 
claims about observable effects. 



Instructions for Identifying Indicators 

1 . Use one claim/indicator worksheet per claim identified. 

2 . Identify the indicators that represent the evidence or documentation that you and your 
community can use to substantiate a claim about the observable effect of your collaborative 
initiative in the long-term. 

3 . Next identify intermediate or short-term indicators that could be used in the next two-years to 
substantiate intermediate effects that will logically contribute to the long-term effects. 

4. In selecting indicators, collaborative teams must strive to identify a number of variables that 
adequately represent the status of children and families in the short- and long-term. At the 
same time, this set must not be so large that the required data cannot be easily collected by local 
collaborative sites (Thorton, Love, & Meckstroth, 1994). 

5 . Begin to talk about existing data sources or the type of instrument that would be needed to 
collect data related to the indicators. In particular, what steps need to be taken regarding the 
identification/use of existing data sources? 
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Issues to Consider in Selecting Indicators 



1 . Does it measure conditions of children, youth, and families at the local level? 

2 . Is it available annually? 

3 . Can trends be tracked over time in ways that show if things are getting better or worse? 

4 . Is the data comparable to data collected in other communities or other similar areas? 

5 . How large is the number of children or families affected (i.e. is the number so small that no 
real conclusions can be drawn from it, as with a change from 2 to 4 children or families, which 
is technically a 100% increase in the problem, but which is an unreasonable interpretation of 
the numbers)? 

6. What data are available now with no increase in spending for data collection or analysis; what 
data are available now but require some analysis to extract; what data are available only if major 
new spending is undertaken? 

7 . Can reasonable estimates be made, even if the ideal data are not available? 

8 . Does the indicator measure the status of children, instead of the scope of the programs (i.e. that 
there are 1000 day care slots in a community may be far less relevant than knowing the 
employment status of working parents or other indicators of the need for child care)? 

9 . Is the indicator skewed by the denominator (i.e. a change in the teen birth rate may have more 
to do with demographic changes than teen behavior and the number of teen births, so watching 
overall population changes is important in interpreting changes in specific indicators)? 
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Sample Claims and Indicators 



Fully implemented collaborative initiatives will produce a variety of outcomes. Identify 
abbreviated claim categories to represent your claim statements about the observable effect of your 
initiative. Next develop a comprehensive list of possible indicators to substantiate it. Be sure to 
identify both short- and long-term effects. The sample indicators below should help you get 
started. The articles included in part five of this manual offer a fuller listing of potential indicators. 



Claim 


Intermediate Indicators 


Long Term Indicators 


(Child and Family Health) 


• Immunization rates 


♦ Incidence of preventable 
diseases 


(Family Functioning) 


• Use of community resources 
and supports 

• Discipline style at home 


• Substantiated incidence of 
abuse or neglect 


(Child Development) 


• Participating in early 
childhood education and 
care 


♦ Milestones in cognitive, 
emotional, social, and 
physical development 


(School Performance) 


• School attendance 


♦ Academic achievement 


(Youth Maturation and Social 
Integration) 


• Employment 

♦ Volunteering 


• Incidence of violent crimes 


(Systemic or Institutional 
Change) 


♦ Improvements in services to 
particular client groups 

• Reduction in costs and 
improvements in efficiency 
of service delivery 

• Increase in use of 
information 

♦ Reduction in barriers 
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Claim and Indicator Worksheet 



Initiative Name: 
Claim: 



NOTE: Most claims will have several indicators and reflect short - and long-term effects. 



Short-term indicators: 



Notes about existing data sources, instrumentation needed, or action to be taken: 



Longer-term indicators: 



Notes about existing data sources, instrumentation needed, or action to be taken: 
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Mapping Key Elements and Indicators 



Definition 

A map of key elements and indicators is a picture of the means and the steps by which the 
collaborative is intended to work. It represents an hypothesis of a sequence of events. A map 
clarifies and systematizes the key elements and indicators of effect that are worth examining. One 
of the advantages of setting down the expected path of change is that some of the assumptions that 
ma y be implicit in the collaborative are made explicit Collaborative planners are able to see the 
link between the key elements of the initiative and the desired outcomes. If long term outcomes do 
not improve, collaborative planners and evaluators have some clue about why. 



Instructions for Mapping Key Elements and Indicators 

In columns from left to right, create lists of your key elements, your short-term indicators, and 
your long-term indicators. Decide which key elements will lead to which short-term indicators. 
Decide next which key elements and/or short-term indicators will lead to which long-term 
indicators. Draw lines to indicate these linkages. 

Since we don’t live in a linear world, there may be overlap. One key element may lead to another 
key element. A key element may lead to multiple indicators. Or, multiple key elements may link to 
a single indicator. 

Ultimately, your map will be complete when each key element or indicator is linked in some way. 

If you end up with key elements or indicators that are not linked in some way, the next step is to 
re-think the specification of key elements and indicators to determine 

• if there is a collaborative desig n issue (e.g., the community identified certain critical 
outcomes that are not related to the collaborative design or certain key elements of the 
collaborative are not related to any valued outcomes); or 

• if there is a documentation or connection issue (e.g. the team did not fully specify all key 
elements or some key elements are not really critical and should be dropped; there should 

be a coherent connection among the elements and indicators). 

As the collaborative initiatives evolve and grow, teams may want to create updated maps of the 
links between key elements and expected short- and long-term outcomes. 
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Mapping Work Sheet 



Initiative Name: 

In columns from left to right, list your key elements, your short-term indicators, and your long- 
term indicators. Decide which key elements will lead to which short-term indicators. Decide next 
which key elements and/or short-term indicators will lead to which long-term indicators. Draw 
lines to indicate these linkages. 

Key Elements Short-term Indicators Long-term Indicators 
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Evaluation Plans 



Definition 

Once the collaborative has defined its key elements, claims, and short- and long-term indicators, 
the next step is to specify a plan for collecting and analyzing data that measures the status of each 
indicator. The major components of an evaluation plan include the specification of the level of 
evidence that will be gathered (outcome evaluations typically focus on measures of impact), a 
sampling plan, the sources of data, instrumentation and data collection procedures, timing of data 
collection, data analysis, and any standards of comparison that will be used as part of the analysis. 
It is in these areas that a local evaluation consultant may be particularly helpful to 
the collaborative. 



Instructions for Completing Evaluation Plans 

Collaborative teams should specify separate evaluation plans for each indicator or groups of closely 
related indicators. Use one evaluation planning worksheet per indicator or set of closely related 
indicators. 

Key components of an evaluation plan include: 

1 . Level of evidence. Decide if the evidence (data) that will be gathered involves 

A. documentation of the level of participation in activities (e.g., percentage of eligible 
population who participate in a service; amount of participation) 

B. gathering perceptions of impact (e.g., self-reports of participants or staff) 

C direct measures of impact (e.g., direct measures of performance) 

If the collaborative team has specified indicators that primarily involve the collection of A and B 
types of evidence, the evaluation design may not yield the type of outcome evaluation that was 
mandated by the Minnesota Legislature. 



2 . Sample. Decide if data collection will include 

A. All potential participants in a specified geographic area 

B. A sample of all eligible participants in a specified geographic area 

C. A specific subgroup (e.g., individuals who actually receive services under the collaborative 
umbrella) 

Collaborative teams need to remember that the overriding interest is in the community as a whole, 
regardless of current levels of participation in various programs or organizations. According to 
Thornton, Love and Meckstroth (1994): 

Data for specific sub populations of the community may misrepresent the status of all families and provide 
a poor basis for policy development. Furthermore, communities are often interested in the extent to which 
families participate in specific activities (such as how often pregnant woman receive adequate prenatal care). 
Such participation rates can only be estimated by looking at all eligible persons, rather than only at those 
using a particular service. 
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At the same time, collaborative teams need to remember that certain indicators are of special interest 
to policy makers (i.e., crime rates and infant mortality), but remain largely impervious to change at 
the community-level. Collaboratives specifying these indicators may want to access data in such a 
way to be able to analyze it separately for participants in collaborative services. Participant data 
might then be compared to data on a community- wide or state-wide level. 



3 . Instrument(s) or procedure(s). Specify if data collection will involve 

A. Assembling data from existing sources either electronically or manually (e.g., birth 
certificates, death certificates, school registration forms, public health surveillance data, 
child abuse reports, school records, police crime reports, census data) 

B. New data collection using an existing instrument (specify) 

C. New data collection involving a specially developed instrument (e.g., surveys of 
community members or participants, direct observation, or direct assessments) 

Collaborative teams will need to determine who will be responsible for assembling data, 
identifying existing instrumentation or developing new instruments, and coordinating any new data 
collection. Teams must balance the value of the information gained from collecting new data with 
the costs of doing so. 



4 . Timing of measurements. Specify the frequency of data collection as 

A. Post-test only — data are collected or assembled once after implementation of a key 
element(s) 

B. Pre-test/Post-test — data are collected or assembled before and after the implementation of a 
key element(s) 

C. Repeated measures - data are collected or assembled periodically (continuously, weekly, 
monthly, annually, every three or five years) 

Collaborative teams will need to determine the actual schedule for collecting or assembling da ta 
Collaboratives should consider initiating repeated measures of long-term indicators now. Initiation 
of collection of data regarding short-term indicators should be tied to the implementation status of 
the key element(s) that are linked to that indicators). If implementation is clearly delayed, data 
collection should also be delayed 



5 . Data analysis. At a minimum specify if indicators will be reported as 

A. Rates 

B. Proportions (for example, the proportion children receiving vision and hearing screening in 
the past 12 months) 

C. Standard scores 

D. Percentiles 

E. Frequencies 

F. Measures of central tendency (e.g. mean, median, mode) 

G . Qualitative Analysis (e.g. coding and reporting on key themes from observation, open- 
ended surveys, focus groups, etc.) 

Note: It is likely that qualitative analyses will be used more frequently as part of local sites' 
internal formative evaluations 




23 



28 



6 . Standard of comparison. It is important for collaboratives to answer the question "As 
compared to what? " regarding any outcome data collected. There are a number of approaches 
for collaboratives to consider: 

A. Comparison to baseline data (e.g., pre-test or data available prior to initiation of 
collaborative) 

B. Comparison to a comparable neighborhood, community, or county 

B. Comparison to statewide data 

C. Comparison to national data 

D. Comparison to data reported in another study that used the same measures 

E. Comparison to an established standard of performance 

F. Comparison of trends over time 
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Evaluation Planning Worksheet 



Initiative Name: 

Specific Indicator(s): 

(check all that apply in each category) 

Level of evidence 

Documentation of the level of participation in activities 

Gathering perceptions of impact 

Direct measures of impact 



Sample 

All potential participants in a specified geographic area (specify below) 

A sample of all eligible participants in a specified geographic area (specify below) 

A specific subgroup (specify below) 



Instrument(s) or procedure(s) 

Assemble data from existing sources either electronically or manually (describe below) 

New data collection using an existing instrument (specify below) 

New data collection involving a specially developed instrument (describe below) 



-) 
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Timing of measurements 

P ost-test only 
Pre-test/Post-test 

Repeated measures 

Other (specify): 



Specify actual schedule for collecting or assembling data. 



Data analysis 

Reported as rate 

Reported as proportion 

Reported as standard scores 

Reported as percentiles 
Reported as frequencies 
Qualitative analysis 
Other (specify): 



Standard of comparison 
Comparison to baseline data 

Comparison of subgroup to eligible population in collaborative as a whole 

Comparison to a comparable neighborhood, community, or county 

Comparison to statewide data 

Comparison to national data 

Comparison to data reported in another study that used the same measure 

Comparison to an established standard of performance 

Comparison of trends over time 

Other (specify): 



Person(s) responsible for instrumentation, data collection, and data analysis: 
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Resources 
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Key Activities of Evaluation Consultants 



Laura Bloomberg and Patricia Seppanen are the evaluation consultants from the University of 
Minnesota, Center for Applied Research and Educational Improvement (CAREI). The work of the 
evaluation consultants includes: 



• Attending meetings of the Evaluation Focus Team and joint meetings of the four 

partners of the Minnesota Children’s Initiative (upon request); 



• Preparing a review of the literature related to family service collaboratives, 
interagency program development, and the evaluation of such initiatives (the current 
draft of this literature review is included in part four of this manual); 

• Articulating an overall evaluation system, including the specification of broad 
evaluation questions; the identification of sources of data to answer each broad 
evaluation question; revision of existing report formats (the quarterly site report 
format and agency reporting strategies); and development of an annual survey form, 
a data collection schedule, and a cross-site data analysis plan; 



• Providing technical assistance and support to the collaborative sites in the 
formulation of their outcome evaluation designs; these technical assistance activities 
will be coordinated with the work of the Children's Interagency Technical 
Assistance Team (ChTTA) and the Evaluation Focus Team; 

• Summarizing evaluation data submitted by the collaborative sites as part of their 
progress reports, an annual written survey, and outcome evaluation reports; as data 
are summarized, the consultants will share it with collaborative sites, the Evaluation 
Focus Team, ChITA, and the Children's Cabinet for discussion; 



• Preparing progress reports and a final report; the progress reports will focus on the 
status of all work to date; the final report will summarize evaluation findings 
regarding Family Service Collaborative and the Minnesota Children's Initiative 
outcomes and impact identified to date. 
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In terms of technical assistance and support, the evaluation consultants are available to 



• Provide two full-day evaluation related workshops for collaborative teams (the 
second workshop will focus on a priority topic identified by the sites such as 
approaches to formative or process evaluation); 

• Review and comment on draft evaluation designs developed by collaborative sites; 

• Participate in conference calls (upon request); 

• Provide individual phone consultation; 

• Identify resource materials to meet common needs expressed by collaborative sites 
(e.g., instrumentation that is developed by sites or lists/copies of instruments 
available from other sources). 



To request technical assistance, contact Laura Bloomberg at 612/625-0502. 

Please fill out and return to Laura the technical assistance request form that appears on the next 
page if your site has immediate needs for assistance. 




35 



34 



Technical Assistance Needs Related to Evaluation 



Initiative Name: 

Contact Person:, 

Phone Number: 

1 . Please indicate below your current needs for technical assistance related to specifying an 
outcome evaluation design. 

( check all that apply) 

— Assistance in specifying key elements of the collaborative initiative 

Assistance in specifying claims and indicators 

Assistance in sample selection 

Assistance in specifying the source of data 

Assistance in selecting or developing instrumentation 

Assistance in specifying data collection procedures 

Assistance in specifying the timing of data collection 

— Assistance in determining appropriate data analysis procedures 
— Assistance in determining an appropriate standard of comparison 



2. Indicate assistance that would be helpful to you in designing other evaluation approaches. 
( check all that apply) 

— Assistance in designing an evaluation of the implementation of collaborative activities 
— Assistance in designing an evaluation of progress on collaborative goals and objectives 
— Assistance in designing an evaluation of the Quality of collaborative processes 
Assistance with other evaluation related work ( specify ) 



3 . Describe the mode(s) of assistance that would be most helpful to you (review of draft 
nMterials, phone consultation, additional training, resource materials such as lists of 
instruments or sample instruments). 
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4 . Summarize any evaluation approaches or materials (particularly data collection instruments) that 
you would be willing to share with other collaboratives. 



Please return this form to: 

Laura Bloomberg 

Center for Applied Research and Educational Improvement 
265 Peik Hall 
University of Minnesota 
Minneapolis, MN 55455 

Phone: 612/625-0502 
FAX: 612/625-3086 
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0 CHAPTER 3 

The Case for Shifting to Resuhs-Based 
Accountability 

By Lisbeth B. Schorr, with the collaboration of Frank Farrow. David Hombeck. Sara Watson 



The current ferment around using results as a way of 
assessing-the success of efforts to improve the lives 
of children and families, and of shifting to a results- 
based accountability system has given rise to hope, 
fear, and confusion. 



Hope that results-based accountability could be the 
key to: 



▲ 




▲ 



▲ 



freeing schools, health care, social agencies, 
and other human services from the rules that 
prevent them from operating flexibly in 
response to the needs of those they serve, 
restoring the public’s faith that both public 
and private human service institutions can 
accomplish their intended purpose, and 
encouraging communities to be more plan- 
ful, more intentional, in how they support 
children and families. 



Fear that results-based accountability will be mis- 
used and bring about: 



A the abandonment of attempts to better the 
conditions of disadvantaged children whose 
effects are difficult to measure or take a long 
time to occur, 

A the erosion of essential procedural protec- 
tions and neglect of concerns for equity, 

A a smokescreen behind which further funding 
cutbacks will be made, or 
A penalizing individual professionals, institu- 
tions, and agencies who may not be 
achieving hoped-for results, but are trying 
hard and doing the best they can. 



Confusion about how to devise and obtain agre< 
ment on a set of goals and outcomes, and reliabl 



ways of measuring results that could justify the 
hopes and quell the fears of the many concerned con- 
stituencies. 

This paper sets out some of the issues in the shift 
to results-based accountability, and identifies a start- 
up list of outcome measures. The authors see 
results-based accountability as an essential part of a 
larger strategy to improve outcomes for children. 
Subsequent materials will be developed to address 
how these outcomes can be translated into a program 
agenda; how that program agenda can then lead to a 
budget and a financial plan; and how, over time, 
results-based accountability can be combined with 
both rewards and penalties, based on performance. 
This paper sets the stage for those later discussions, 
but all readers, especially policymakers, should note 
that the real benefit of results-based accountability 
can be realized only through that full sequence of 
activities. 

I. What is at Stake in the Shift Toward Results- 
Based Accountability 

A. Results-based accountability can replace — or at 
least diminish the need for — centralized bureau- 
cratic micromanagement and rigid rules. Effective 
services require a significant degree of both local 
variation and frontline discretion, which cannot be 
maintained in the face of detailed regulation of pro- 
gram inputs that tie the hands of front-line profes- 
sionals. Management by results is the best alterna- 
tive to top-down, centralized micromanagement, 
which holds programs responsible for adhering to 
rules that are so detailed that they interfere with a 
program’s or institution’s ability to respond to a wide 
range of urgent needs. 

Policy makers can be encouraged to desist from 
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P olicy makers ccn be er.ccurcgec 'a 
desist from regulating inputs cnc 
prescribing detailed procedures if they 
have the capacity to hold programs, 
institutions, and those who implement and 
run them accountable for results. 



regulating inputs and prescribing detailed procedures 
if they have the capacity to hold programs, institu- 
tions, and those who implement and run them 
accountable for results. The use of outcome indica- 
tors that reflect common sense and common under- 
standing (indicators that show, for example, that the 
rates of low-weight births are being reduced, or that 
more students are demonstrating age-appropriate 
mastery of school skills) helps to focus attention on 
agency mission rather than rules, and permits the 
necessary flexibility and autonomy at the front-end. 
Auditors can spend less time reviewing records to 
see how many services were provided (e.g., how 
many families were seen) and whether the forms 
documenting eligibility for services were properly 
filled out. Instead, they spend more time on inquir- 
ing into the results achieved (such as the reduction of 
multiple or inappropriate out-of-home placements 
for children). The question asked of professionals at 
the front-lines, be they teachers, social workers, or 
health professionals, would shift from “Did you do 
what they told you to do?” to “Did it work? What 
difference did it make in outcomes for children?” A 
changed organizational climate would be the result, 
in which well-trained professionals would be able to 
use their judgment and experience to respond flexi- 
bly to the needs of children and families, rather than 
being constrained by pressures of funders who see 
detailed input regulation as their only means of pro- 
tecting the public interest. 

One example of how this approach can work 
comes from Kentucky: under the Kentucky Educa- 
tion Reform Act, the state decides what , and local 
communities decide how . (The process of deciding 
what a state will hold communities accountable for 
will optimally involve extensive consultation, as we 
discuss in Section V.A. on page 22.) 



B. Outcome information can assure funders and the 
public that investments are producing results. Fun- 
ders and the public are demanding information on 
which informed judgments can be made about 
whether institutions, programs and policies are in 
fact accomplishing their intended purpose. Polls 
show that a public that has lost confidence in govern- 
ment and other institutions is prepared to support 
new investments in schools and services when citi- 
zens are convinced that the investment is bringing 
promised results. Especially in a time of fiscal con- 
straints, programs and agencies willing to be held 
accountable for achieving agreed-upon results will 
have a greater chance of obtaining needed funding 
and other support. 



C. Agreement on desired results can facilitate cross- 
systems collaboration on behalf of children. As 
more individuals and agencies from different sys- 
tems, disciplines, and backgrounds attempt to work 
together, they encounter barriers that are difficult to 
overcome in the absence of shared purposes that are 
explicitly defined and agreed upon. Agreement on a 
common set of goals and outcome measures not only 
makes collaboration easier, but also helps promote a 
community-wide “culture of responsibility” for chil- 
dren and families, and fuels the momentum for 
change. 



D. Agreement on desired results helps to minimize 
investment in activities that don’t contribute to 
improved results. Reflecting Alice in Wonderland’s 
insight that if you don’t know where you’re going, 
any road will get you there, a focus on outcomes is 
likely to discourage expenditures of energy, political 
capital and funds on empty organizational changes 
and on ineffective services. Community-wide accep- 
tance of shared outcomes thus helps to put service 
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A focus on results forces the question or 
whether outcome expectations must 
be scaled down, or interventions and 
investments scaled up. It assures attention 
to the adequacy question. 



integration-efforts into proper perspective. Fragmen- 
tation is such a big problem in existing services, that 
a great deal of energy is currently going into efforts 
to link services together. But while collaboration is 
a necessary condition for providing effective ser- 
vices, it is not sufficient. Alone, it will not improve 
results. It is futile to put together services that are 
ineffective because they are of mediocre quality, are 
rendered grudgingly, are rendered by professionals 
who don’t work respectfully with families, and are 
unable to respond to the unique characteristics of the 
community they serve. The shared commitment to 
improve outcomes for children is what can make 
efforts at collaboration fall into place — not as an 
end, but as an essential means of working together 
toward improved results. 

E. Information about results enhances community 
and agency capacity to judge the effectiveness of 
their efforts, and to modify activities in response to 
impact information. Because this is a time of rapid 
change in education, welfare, health and human ser- 
vices practices, programs, and systems, it is particu- 
larly important to improve the ability to make judg- 
ments about the effects of change: whether new or 
reformed services, new relationships among service 
providers and between providers and communities, 
and new connections among schools, services, and 
other community and economic development efforts, 
are working and worth the investment. When 
results are tracked on a timely basis and connected to 
service strategies, outcome information can provide 
guidance to program development as initiatives 
evolve. 

F. A focus on results clarifies whether allocated 
resources are adequate to achieve the outcomes 
expected by funders and the public. An outcomes 



focus highlights how much investment is required to 
produce significant improvement in the lives of chil- 
dren and families. The new conversation about 
results is promising (or threatening) to end a conspir- 
acy of silence between funders and program people. 
It exposes the fact that human service providers, edu- 
cators, and community organizations are consistently 
expected to accomplish massive tasks with trivial 
resources and inadequate tools. A focus on results 
forces the question of whether outcome expectations 
must be scaled down, or interventions and invest- 
ments scaled up. It assures attention to the adequacy 
question: are the resources being committed ade- 
quate to achieve the intended purpose. 

A parent education program may be expected to 
reduce the incidence of child abuse in a neighbor- 
hood although it consists of only a few didactic 
classes. An outreach program to get pregnant 
women into prenatal care may be expected to reduce 
the incidence of low birthweight in a community 
although the sources of prenatal care are overcrowd- 
ed, impersonal, and have no capacity to deal with 
homelessness, drug abuse, or lack of social support. 
In the past, when a significant discrepancy occurred 
between aspirations and documentable (or even 
potential) accomplishments, the response has typical- 
ly been to retreat from a commitment to changing 
outcomes. As program managers and funders face 
up to the difficulties of actually changing real-world 
outcomes, and recognize the relative weakness of 
single, underfunded, piecemeal interventions com- 
pared to powerful social, economic, and demograph- 
ic forces that push outcomes in a negative direction, 
they often agree just to measure how many people 
were reached or how attitudes and knowledge have 
changed among program participants. Together, all 
parties agree that it is unfair or inappropriate to hold 
the program accountable for actually improving 
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results when the program is doing the best it can. 
The reality that the investment is insufficient is there- 
by obscured. 

In circumstances where it will take a critical 
mass of comprehensive, intensive, interactive inter- 
ventions to change outcomes in a defined population, 
where interventions must be able to impact even 
widespread despair, hopelessness and social isolation 
in order to be effective, it is counterproductive to 
hide from ourselves the limitations of most current 
efforts. Recognition that the addition of a single new 
intervention, or even recognition that services alone 
may not be sufficient to change outcomes in the 
absence of changes in employment, economic securi- 
ty, physical safety and housing, is not an argument 
against results-based accountability. It is an argu- 
ment for adequate funding of promising interven- 
tions, and an argument for human service agencies 
and schools to forge partnerships with efforts to cre- 
ate jobs and to improve housing and public safety, 
and to re-create the infrastructure of communities. It 
is an argument for defining what the community con- 
siders a set of adequate outcomes and then making 
sure that the resources needed to reach those results 
are made available. 

II, Risks — to Recognize and to Minimize — in 
Making the Shift to Results-Based Accountability 

While the shift toward results-based accountabil- 
ity brings many clear advantages, it is not an 
unmixed blessing. In fact, it carries real risks, which 
must be recognized and minimized through careful 
strategizing and thoughtful implementation. 

A. Funders , (he public, and even program people , 
may underestimate how long it takes to achieve sig- 
nificant improvement in outcomes . The single most 



frequently cited lesson from major current reform 
efforts is that it takes much more time than expected 
— both to get the initiative under way, and to get it to 
the point where it begins to show an impact on real- 
world outcomes. 

B . Demands for documented results could drive pro- 
grams away from achieving their mission of improv- 
ing a broad range of results for a broad target popula- 
tion, when agencies: 

▲ engage in “creaming” and ducking hard 
cases, or 

▲ distort their activities to emphasize those 
that will show measurable and rapid results 
while neglecting equally or more important 
efforts that are harder to quantify and whose 
results don’t show up as quickly. (For 
example, if immunization rates are the out- 
come on which health services are judged, it 
will be important to guard against allowing 
an emphasis on immunization strategies to 
result in the neglect of other primary and 
preventive services.) 

C. Funders could be led to confine their support to 
interventions whose effects are readily and quickly 
quantifiable , in preference to more effective or sub- 
tler interventions whose benefits are harder to docu- 
ment. (A church-sponsored children’s choir or a 
recreation center for teenagers may add to a sense of 
community, and thereby contribute to improved out- 
comes, without being able to demonstrate a direct 
outcome impact.) Protection must be explicitly pro- 
vided to interventions that are vulnerable because: 

▲ their effects show up years later, 

▲ the data to document their impact are not 
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available, or the technology to measure 
their effects does not exist, or 
▲ the intervention is one of many preventive 
efforts that interact to produce cumulative 
later effects. 

D. Some forms of outcome measurement could lead 
to labeling and stigmatizing children. Efforts to 
assess school readiness, for example, must be 
designed to take into account that “readiness” does 
not inhere solely in the child, that children don’t all 
develop at the same rate, and that “readiness” is not 
just a cognitive matter, but includes physical and 
emotional health and social competence. In addition, 
school readiness should be assessed on a sample 
rather than an individual basis, to avoid labeling of 
individual children. 

E. Agency accountability could be weakened as 
attention shifts to community-wide accountability 
efforts. Achieving any of the core outcomes requires 
the activities of multiple agencies, as well as infor- 
mal supports and community activities that are 
beyond a single agency’s control. Since no one agen- 
cy, acting alone, can achieve any of the agreed-upon 
results, the question arises, how should individual 
public agencies, or systems — for example, the child 
welfare agency or the mental health system — hold 
themselves or be held accountable. 

The answer lies in a careful “unbundling” of 
each outcome into the detailed steps and strategies 
which, together, can produce the intended result. For 
example, to increase healthy births, a host of related 
actions are necessary, one of which is likely to be 
that the local health department must organize and 
conduct outreach to all high-risk mothers to ensure 
that they have adequate prenatal care. Once the 
health department’s role is specified within the con- 
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text of the overall strategy, measures can be applied 
to how well the health department is performing this 
role and the department can be held (and can hold 
itself) accountable for this result. So, to continue this 
example, if all parts of a strategy to increase healthy 
births are successfully accomplished except one 
(e.g., the health department’s), then the health depart- 
ment can be held accountable by the community for 
better performance. 

F. The shift to results-based accountability should 
not be seen as a panacea . An outcomes orientation 
does not guarantee the design of an effective com- 
munity service and support system. Especially 
because current understanding of the precise connec- 
tions between specific interventions and specific out- 
comes is limited, an outcomes orientation does not 
solve the problem of identifying needed and missing 
elements of a comprehensive approach to improving' 
results. However, by informing and focusing the 
process, result-based thinking does seem to support 
the development of a culture that is less rule-bound 
and more mission-driven. 

G. The shift to results-based accountability cannot 
be allowed to substitute for rock-bottom safeguards 
against fraud , abuse , poor services, and inequities or 
discrimination based on race , gender ; disability or 
ethnic background. Supporters of improved out- 
comes for children may in the past have overestimat- 
ed the extent to which equity and quality could be 
safeguarded by regulation, and underestimated how 
much detailed mandating and strict rules could 
undermine responsiveness, flexibility, and the exer- 
cise of discretion. If the shift to results-based 
accountability is to attain its promise, it must be 
combined with better training of frontline workers 
and managers and careful monitoring, to make the 
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achievement of less rigidity in the provision of ser- 
vices compatible with the protection of vulnerable 
children and families. 

The new outcomes orientation should not lead to 
the abandonment of all input and process regulation. 
Procedural protections will have to be maintained to 
protect against fraud, poor services, and inequities or 
discrimination based on race, gender, disability, or 
ethnic background. Procedural protections will also 
have to be maintained where there is no other way to 
restrict the arbitrary exercise of frontline discretion 
by powerful institutions against the interests of pow- 
erless clients. At the same time, reliance on process 
measures should not be allowed to drive out or 
obscure a continuing focus on results-based account- 
ability. 

H. When a community adopts a results-oriented sys- 
tem, fundamental changes in the allocation of 
resources will result and must be prepared for. 
One key indicator of whether the outcomes orienta- 
tion is influencing agency performance will be 
whether budget priorities change in response to the 
identification of shared outcome measures. The 
shift to results-based accountability will be of little 
use in a community if it does not translate into man- 
agement and budget priorities. The adoption of com- 
munity outcomes will be ineffective if agencies 
revert to tradition and divide the results into isolated 
subsets that would be pursued separately by schools, 
social service agencies, the health sector, mental 
health agencies, juvenile justice, and others. As 
agencies, communities and systems begin to manage 
their resources to achieve agreed-upon results, set 
priorities based on these outcomes, and are held 
accountable for making progress toward these out- 
comes, some agencies will actually lose resources in 
the process. That risk must be acknowledged and 
prepared for at the outset. 



III. Community Goals and their Relationship to 
Results-Based Accountability 

Communities, states, and collaboratives can best 
begin the results-based accountability process by 
asking the question, “What do we want for our chil- 
dren?” By beginning in this way, these deliberations 
are usually able to go beyond earlier project- or prob- 
lem-bounded discussions. They are then more likely 
to produce a consensus around basic goals which 
together constitute a vision that can serve to: 



▲ capture aspirations, 

▲ unify people who differ in many other ways 
around common aims for their children, 

▲ focus attention not only on a wide range of 
formal services, but also on informal sup- 
ports and opportunities, and 

▲ surface value issues in the context of what 
people want for their children, before they 
get into the question of what is to be mea- 
sured. 



Examples of the goals that communities might 
choose include: 



A higher rates of healthy births, 

A higher rates of children ready for school 
learning at the time of school entry, 

A higher rates of children succeeding at 
school, and 

A higher rates of youth who achieve personal, 
social, and vocational competence (which 
might be further broken down into higher 
rates of youth who feel safe, who have a 
sense of self-worth, a sense of mastery, a 
sense of belonging, a sense of personal effi- 
cacy, who are socially, academically, and 
culturally competent, and who have the 
skills needed for productive employment ). 1 
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As a community embarks on the job of building 
an effective system of services and supports, these 
goals can become a framework that can be used for 
many purposes — inspirational, guiding service 
delivery, and guiding the selection of outcome mea- 
sures for accountability purposes. 

Only some aspects of these goals can currently 
be measured with widely available data and with out- 
come measures around which it is possible to gain 
widespread agreement. Most communities will have 
aspirations for their children that exceed the results 
that are currently measurable. Goals and outcome 
measures serve different purposes. The former rep- 
resent what the community is striving for. The latter 
represent what the community will be held account- 
able for — by public and private funders and perhaps 
by higher levels of government. The goals can be 
general, but the outcome measures must be so specif- 
ic, the public stake in their attainment so clear, and 
their validity and reliability so well established, that 
the community would be willing to see rewards and 
penalties, as well as resource allocation decisions, 
attached to their achievement . 2 

Thus the list of outcome measures that is 
attached represents minimal rather than maximum 
objectives, with the advantage of: 



A allowing measurement to begin, 

A allowing communities to reach agreement 
on results for which they are willing to be 
held accountable, and 

A allowing voters, professionals, and program 
participants to see the direct connections 
between the goals for children they are try- 
ing to achieve, and the outcome measures 
used for accountability purposes. 




We believe that a commitment to more visionary 
goals is entirely compatible with a commitment to 
documenting progress toward the achievement of 
these goals by the use of the outcome measures we 
propose. Agencies and communities that align their 
activities with these measurable outcomes (i.e., 
“teaching to the test”) would be moving dramatically 
in the right direction. Furthermore, and most impor- 
tant, while the outcome measures we propose stop 
short of a more ideal and ambitious set of goals, sub- 
stantial progress toward these outcomes would signi- 
fy a radical improvement in the prevailing conditions 
of children and families. 

IV. Starting the Shift to Results-Based Account- 
ability 

We believe that it is critical to start implementing 
systems of results-based accountability now, recog- 
nizing that the necessary procedures and technolo- 
gies are still in the process of development. Too 
much is at stake not to begin. Not only is the overall 
well-being of children deteriorating in many ways, 
but the credibility of government capacity to help 
improve these outcomes is at its lowest point ever. If 
we are to maintain, let alone expand, investments in 
children’s futures, we must be able to produce mea- 
surable results. 

Although there are still many problems to solve 
in moving towards holding systems accountable on 
the basis of results achieved, we believe that the 
potential usefulness of results-based accountability 
now outweighs the risks. We believe that the shift to 
results-based accountability must be made carefully 
and thoughtfully, and that it must be led by those 
who care about both the process and the results, and 
not left to those who find it easy because they don’t 
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understand the issues. 

Many communities and agencies around the 
country have come to similar conclusions, and have 
already been hard at work, defining the goals and 
outcomes they intend to pursue. We have tried to 
learn from these processes now under way, and have 
identified a list of Child and Youth Outcomes, 3 
around which it is relatively easy to obtain broad 
agreement, and about which data are readily avail- 
able. We believe that the process of agreeing on a set 
of outcome measures will be simplified and speeded 
up if every state, every local community and every 
new initiative didn’t have to start from Square One. 
The list of outcome measures, shown on the chart on 
page 21, is intended to serve as a starting point for 
discussion and negotiations. 

For each of the outcome measures on our list, we 
also have compiled the following information (to be 
found in a separate volume, “A Start-up List of Out- 
come Measures With Annotations,” available 
through the Center for the Study of Social Policy): 

▲ definition 

▲ significance 

▲ relevant facts 

▲ national, state, and local data sources 

▲ additional comments 

▲ related measures. 

Our list is limited to child and youth outcomes, 4 
follows a developmental sequence (from birth 
through childhood to young adulthood), and is based 
on criteria specifically designed to produce a set of 
outcomes for immediate use. We present our criteria 
and our rationale for inclusion here so that they can 
stimulate the discussion that is essential if a results- 
based orientation is to be widely adopted. 
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A. A start-up list should consist of outcome mea- 
sures that are considered important and meaningful 
by a wide range of policy makers, funders, and citi- 
zens. In our view, the results chosen for account- 
ability purposes should be transparent measures of 
success that are persuasive to skeptics, not just to 
supporters of the programs and policies being 
assessed or held accountable. (For example, skeptics 
are more likely to be impressed by changes in the 
rate of unmarried teenage childbearing than by 
changes in self-esteem scales.) 

B. A start-up list should consist of outcome mea- 
sures about which data are relatively easy to obtain, 
primarily from existing official data sources, and to 
interpret. For example, outcomes such as the appro- 
priate receipt of special education services are not 
included because the data are difficult to gather for 
large numbers of children and hard to interpret. 

C. The distinction between outcome measures and 
process (or capacity) measures should be clearly 
maintained. Indicators that measure processes and 
capacities (rather than results) should be included as 
a significant part of a results-based accountability 
effort when they are clearly related by empirical evi- 
dence to significant outcomes. (For example, early 
receipt of high quality prenatal care has been shown 
to be clearly related to positive birth outcomes; on 
the other hand, participation in didactic parent educa- 
tion classes, even when it results in increased knowl- 
edge, has not been shown to improve parenting or 
decrease child abuse or neglect.) When process 
measures (e.g., school attendance, formation of a col- 
laborative) or capacity measures (e.g., the availability 
of high quality child care) are used as part of results- 
based accountability, the rationale for using them 
instead of outcome measures should be clearly 
understood and stated. 
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CHILD AND YOUTH OUTCOMES 

A Core List to Serve as a Starting Point 



Healthy Births 

Lower rates of low birthweight births 

Higher rates of early prenatal care 

Lower rates of births to single mothers under 1 8 

Two- Year-Olds Immunized 

Children Ready for School 

Immunizations complete 

No untreated vision or hearing problems 

School-readiness traits as identified on sample basis 



Children Succeeding in Elementary, Middle, and High School 

As indicated by lower rates of school drop-out, and by 

Academic achievement measures demonstrating competency over challenging 
subject matter in grades 4, 8 and 1 2 

Children and Youth Healthy, Safe, Prepared for 
Productive Adulthood 

Children not abused or neglected 
Children living in own family 

Children living in families with incomes over the poverty line 
Youth avoiding 

Early unmarried childbearing 

Substance abuse 

Arrests for violent crime 

Suicide 

Homicide 

Accidental death 

Sexually transmitted diseases and HIV/AIDS 
Young adults in school or employed 
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tors that could measure short-tetm steps toward long- 
term outcomes and goals. 
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measures, so that communities and agencies can 
make sure they do not slip back into reliance on the 
input measures that results-based accountability was 
meant to replace. 

D. Is there a role in results-based accountability for 
cost-savings outcomes? One outcome measure that 
has proven useful — even though it is a departure 
from child and youth outcome measures — is an 
indicator of cost-savings that occur as a result of the 
provision of improved services. Although many 
important savings occur in budgets and at times that 
are far removed from the new or improved service, 
in instances when the savings can be directly attribut- 
ed to the intervention, the evidence of cost-effective- 
ness can be dramatic and persuasive. For example, 
high quality prenatal care and supports have been 
shown to save several times their cost in neo-natal 
intensive care. We advocate using cost-savings mea- 
sures as a supplement to individual outcome mea- 
sures rather than as a substitute. 

E. What is the relationship between results-based 
accountability and evaluation research? The role of 
the evaluator and of evaluation research would 
undergo significant change in a world in which 
results-based accountability were the norm and not 
the exception and in which outcomes have become 
an important part of the everyday way that people 
think about programs, policies, and reform initia- 
tives. This is especially true with respect to the eval- 
uation of complex, multi-system, multi-disciplinary 
interventions that are expected to impact children, 
families, institutions, and whole communities. 

First, when desired outcomes are specified by a 
community or state as part of its resource allocation 
or reform efforts, it is reasonable to base the impact 
evaluation of the program or community initiative on 
those outcomes. Then the evaluator would no longer 
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assume the responsibility for selecting (or negotiat- 
ing with program people to select) the outcomes on 
which the effort will be evaluated. In addition, since 
the outcome measures used for accountability pur- 
poses tend to be ones which are already being col- 
lected for other purposes, usually by official agencies 
(e.g., rates of low birthweight, immunization, 
arrests, school completion, employment, etc.), they 
are more likely to be widely recognized for their 
“real world” significance than are measures that 
show performance on scales constructed by 
researchers primarily for research purposes. The use 
of outcome measures that are already being collected 
for other (usually public) purposes has the effect of 
vastly reducing evaluation costs, whether measured 
in money, time, intrusiveness, or required expertise. 
Outcome measurement would then no longer rely 
solely on an expert intermediary, on whom program 
managers, funders and communities are dependent 
for information as to whether the program is suc- 
ceeding. The result could be a demystification and 
democratization of the outcomes evaluation process, 
encouragement of citizen monitoring of the welfare 
of children and families, greater access to outcome 
evaluation findings, and an increase in usable knowl- 
edge. 

Second, when the impact of an initiative on 
selected results is measured for purposes of account- 
ability, it is possible to draw a greater distinction 
between the two evaluation functions of assessing 
results (i.e., documenting the extent to which agreed- 
upon results were achieved), and of attributing causal 
connections (i.e., making judgments about the pro- 
cess by which those results were obtained). If evalu- 
ators were free to focus more on the ‘why’ and ‘how’ 
(as opposed to the ‘whether’), they could position 
themselves to better obtain an accurate and nuanced 
understanding of the nature of interventions, and to 
provide an accurate description of the interventions, 
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because they would not have to maintain the tradi- 
tional distance between evaluator and provider that 
has been required to provide the evaluator with an 
“outsider’s” credibility. 

Evaluators who are no longer responsible for 
answering whether the initiative worked, but can 
focus on how and why it is working, can move 
toward an explicit stance of helping practitioners to 
become more reflective, to extract theory from their 
daily experience, to learn contemporaneously from 
their experience, and thereby to improve their prac- 
tice and their intervention. The new evaluator could 
provide feed-back to practitioners for mid-course 
corrections, enhancing their capacity to reflect and 
do on-the-spot experimentation. Evaluators could 
help practitioners to think more carefully about both 
the theory and practice of what they’re doing. They 
could develop a new evaluation culture that would 
lead to a greater understanding of successful inter- 
ventions and how results are related to interventions. 
Evaluators would become collaborators of reflective 
practitioners in the interest of program improvement, 
and of helping others to learn from proven and 
promising interventions. 

VI. Next Steps 

A. Continuing work on making a start-up list more 
useful. We expect to continue work on helping com- 
munities and programs to implement results-based 
accountability. We expect that as communities and 
agencies begin using results-based accountability 
tools, they will need additional information and 
advice on such matters as: 

▲ how to get the necessary information, 

▲ which measures are most appropriate with 
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what size populations, in which specific cir- 
cumstances, and over what period of time, 

▲ improving the tools that will make it possi- 
ble to measure outcomes in useful units 
(such as school catchment areas, neighbor- 
hoods, census tracts, zip codes, etc.), 

▲ how changes in outcomes should be under- 
stood in relation to background factors (such 
as how to account for changes in outcomes 
that might be attributed to the closing of a 
factory or a sudden influx of new immi- 
grants), 

▲ how to select appropriate comparisons 
against which to measure outcomes, includ- 
ing the use of comparisons over time, com- 
parisons with groups outside the communi- 
ty, and comparisons among various racial 
and income groups, 

▲ how to avoid misuse of the chosen outcome 
indicators, 

▲ how long it takes before it is reasonable to 
expect change in a given measure, and the 
magnitude of change that can be expected in 
relation to the size and nature of the problem 
and of the population, 

▲ understanding options with regard to attach- 
ing rewards and penalties to the achievement 
of agreed-upon results, and 

A identification of reliable short-term indica- 
tors that could measure small steps toward 
ultimate goals. 

B. New work to increase the significance of what is 
being measured. Several of the outcome measures 
we recommend are not a perfect match for the out- 
comes we are recommending that communities get 
information about. For example, the indicator “Chil- 
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dren living in their own family” is an imprecise mea- 
sure of the outcome we are really after, which is 
“Children living safely in their own homes or in sta- 
ble out-of-home care.” We will continue to seek 
more reliable approximations, such as “Children who 
have experienced multiple out-of-home placements.” 
The school-readiness outcome poses special mea- 
surement problems, which we expect to work on 
with communities. 

Work should also proceed to identify new ways 
of achieving a closer match between outcome mea- 
sures and long-term goals, by measuring additional 
widely acceptable indicators of positive well-being, 
such as responsible citizenship, healthy and nurtu- 
rant family functioning, and healthy parent-child 
interactions. Consideration should be given to 
whether a major investment, on the order of the New 
Standards Project , 6 is called for to achieve this pur- 
pose. 

C. New work on identifying intermediate indicators. 
The many funders, practitioners, managers, and sys- 
tems reformers who are adopting an outcomes orien- 
tation increasingly recognize that they must obtain 
information about results during time periods that are 
meaningful to politicians, and that means relatively 
quickly — often long before a program is “proud,” 
and long before it has had a chance to make an 
impact on rates of school readiness, child abuse, 
school success, teenage pregnancy or violence. Sus- 
tained work is needed to produce reliable, short-term 
indicators that could measure initial progress toward 
ultimate goals. 

There are two primary ways to assess progress 
toward the achievement of agreed-upon outcomes 
over the short-term: One is by measuring interim 
outcomes, and the other is by measuring the commu- 



nity’s capacity to achieve the identified long-term 
outcomes. 

New work on measuring interim outcomes. 
More work is needed to test hypothesized connec- 
tions between short-term and long-term outcomes, 
although some connections seem to be fairly well 
established. For example, an improvement in school 
attendance rates is thought to predict an improve- 
ment in school achievement rates. In a current evalu- 
ation of family preservation, such indicators as par- 
ent’s sense of mastery, social support, and parental 
substance abuse are used as interim measures to pre- 
dict such long-term outcomes as the recurrence of 
abuse or neglect. Such process measures as extent of 
participation, client satisfaction, and an increased 
sense of community may also come to be seen as 
reliable precursors of improved results. 

New work on identifying indicators of communi- 
ty capacity. The biggest obstacle to identifying indi- 
cators of a community’s capacity to reach long-term 
outcomes is that little is known about the precise 
relationships between the components of capacity 
and outcomes. The identification of indicators of 
community capacity that could guide communities in 
their efforts to reach long-term outcomes depends on 
having or developing reliable theories — or at least 
sturdy hypotheses — about the relationship between 
interventions and results, and about the constellation 
of conditions and interventions that will lead to good 
results. For example, when a community is develop- 
ing strategies to reduce rates of low weight births, 
can a funder or governing entity say with confidence 
that the “enabling conditions” to reach that outcome 
are the capacity to provide high quality, responsive 
prenatal care, nutrition services, and family support 
to pregnant women, and family planning services to 
all persons of child-bearing age? If the outcome in 
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question were school readiness, is it possible to say 
that higher rates of school-readiness will be achieved 
if the community develops its capacity to provide 
high quality child health care, family support ser- 
vices, child care, Head Start, nutrition services, etc.? 

There is little consensus on the constellation of 
services and supports that must exist in a community 
that can be reasonably assumed to constitute the pre- 
conditions for improved results, especially since 
many interventions rely on their interaction with 
other interventions for their effectiveness. For exam- 
ple, a high quality after-school program is probably 
effective only in combination with many other ser- 
vices and supports that could together make a differ- 
ence. Furthermore, it is not enough to know of the 
simple existence of certain services, one would have 
to take their quality and other attributes into account. 
The distinction between service availability and the 
nature and quality of the service in accounting for 
improved results is absolutely crucial — and requires 
much greater understanding than now exists. Ulti- 
mately, theoretical understanding might even allow 
one to take account of certain preconditions, such as 
an incentive structure to support desirable services, 
the availability of certain kinds of training and pro- 
fessional development activities, or the provisions 
made to respond in a culturally sensitive way to a 
variety of populations. 

In some areas, existing empirical evidence per- 
mits clear connections to be made between inputs 
and results. For example, the work of Joyce Epstein 
and colleagues at Johns Hopkins University 7 has 
shown that schools can enable more families to 
become and stay involved in their children’s educa- 
tion, and that parent assistance at home has important 
consequences for children’s achievement, atten- 
dance, school adaptability, and classroom behavior. 
In the main, however, much systematic empirical 



work is required before short-run measures of 
capacity can reasonably be related to long-term out- 
comes. For example, the availability of prenatal care 
and health insurance are surely related to improved 
birth outcomes, but whether the relationship is strong 
enough, and whether their effect on outcomes is 
actually a function of their availability (rather than of 
their quality), so that their availability can be used as 
an interim indicator, is an open question. 

Little is also known, though much is hypothe- 
sized, about the connection between indicators of 
community-level change, and outcomes for children 
and youth. It is reasonable to assume that a neigh- 
borhood that is building an infrastructure of informal 
supports and economic opportunity is likely to lead 
to better outcomes for children, but there is little 
agreement on ways to measure community building 
and economic opportunity, and little understanding 
of the precise connections. 

In primary and secondary education, the 
response to demands for interim measures has taken 
the form of what have come to be known as “Oppor- 
tunity to Learn Standards,” or “School Delivery 
Standards.” The rationale for these standards is that 
it would be unreasonable to expect students to per- 
form at world class levels in science if the school 
they go to has no chemistry or biology labs, and that 
the presence or absence of chemistry and biology 
labs would be something the community could look 
at to assess its progress toward desirable results, 
more simply and at an earlier time than measurable 
changes in achievement could be expected to occur. 

The downside of the extensive application of 
capacity measures to assess progress toward long- 
term outcomes, of course, is how closely they could 
come to resemble the input regulation that results- 
based accountability was meant to replace. 
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VII. Conclusion 

This paper has discussed the case for results- 
based accountability, and has proposed a start-up list 
of outcome measures that could be used by commu- 
nities as they begin to implement this approach. 
Agreement on outcome measures, however, is just 
the first step in installing a community-wide 
approach to service design, delivery, and account- 
ability that is rooted in results. The key next step is 
for the community to use results to develop an effec- 
tive program agenda. That step, in turn, must be 
complemented by attention to the collaborative gov- 
ernance strategy that maintains the results-based 
accountability; by the financial strategy for funding 
the program agenda; and by the professional devel- 
opment and training strategy that can help build 
frontline and administrative staff skills to support 
this new approach. Those topics are the subject of 
future materials from the Improved Outcomes for 
Children Project. 

Notes 

1. Pittman, K. and S. Zeldin. Evaluating Youth Development in 
Programs and Communities: The Need for an Integrated 
Framework and Collaborative Strategy. Washington, D.C.: Cen- 
ter for Youth Development and Policy Research: 1992. 

2. Especially when it comes to adolescents, many communi- 
ties and many programs will be working to achieve goals that go 
well beyond what they are able to measure. We believe that a 
programmatic commitment to achieving such goals as higher 
rates of youth who have a sense of self-worth, a sense of mas- 
tery, a sense of belonging, and a sense of personal efficacy, is 
not incompatible with accountability systems that rely on out- 
comes that are more easily measured and that may be more per- 
suasive in a public policy context because their harm is widely 
agreed upon (e.g., reducing the rate of school drop out, teenage 
pregnancy, and juvenile crime). 

There are many who believe, with Albert Shanker, Presi- 
dent of the American Federation of Teachers, that the opposition 
to Outcome Based Education is the result of over-reaching. 
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Shanker cites the following examples of outcomes chosen by the 
States of Pennsylvania and Ohio which he believes were too 
ambitious and too ambiguous to common widespread accep- 
tance: 

A All students demonstrate a comprehensive under- 
standing of families, their historical development, and 
the cultural, economic, social, and political factors 
affecting them. 

A All students learn to function as responsible family 
members. 

A All students maintain physical, emotional, and social 
well-being. 

A All students learn to establish priorities to balance 
multiple life roles. 

3. This list is based on our work with site partners in the 
National Alliance for Restructuring Education, the earlier work 
of the Joining Forces effort of 1990-92, and on a survey by the 
Center for the Study of Social Policy of outcomes measures in 
current use by cross-systems community reform efforts and 
advocacy groups, including Kids Count and Children Now. 

4. Obviously our focus on children and youth is only one of 
the many possible choices in the shift to outcomes accountabili- 
ty. It would be theoretically possible to expand one’s focus to 
include family and community outcomes. Whether a given indi- 
cator measures an outcome, an input, or a crucial background 
factor that must be considered in interpreting outcomes depends 
on the goals of the effort. For example, the availability of 
affordable housing could be an outcome of a housing program, 
an input in a community development program, or a background 
factor in assessing a children’s intitiative. 

5. Young, Gardner, and Coley, Chapter 2. 

6. The New Standards Project is a consortium of states and 
cities developing high academic achievement standards, and 
new valid, reliable assessment instruments to measure them. 

7. Epstein, Joyce L. School and Family Partnerships. In: 
Encyclopedia of Educational Research, Sixth Edition. M. Alkin 
(Ed.) NY: Macmillan: 1992. 
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I. MEASURING THE STATUS OF CHILDREN ANI) FAMILIES 1 



When you can measure what you are speaking about, and express it in numbers, you know 
something about it; but when you cannot measure it, when you cannot express it in numbers, 
your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, 
but you have scarcely, in your thoughts, advanced to the stage of science. 

William Thomson, Lord Kelvin 
Popular Lectures and Addresses 



Grown-ups love figures. When you tell them that you have made a new friend, they never ask 
any questions about essential matters. They never say to you, "What does his voice sound like? 
What games does he love best? Does he collect butterflies?" Instead, they demand: "How old 
is he? How many brothers has he? How much money does his father make?" Only from these 
figures do they think they have learned anything about him. 

Antoine de Saint-Exup6ry 
The Little Prince 



As communities across the country continue to grapple with ways of improving the lives of 
children and families, their desire for community-level numerical measures has increased. These 
measures provide a basis for understanding the problems children and families face and for assessing 
efforts to improve their circumstances. Communities want to know such things as how many mothers 
obtain adequate prenatal care, how many babies are born healthy, the number of two-year-olds who 
are fully immunized, whether children enter school ready to learn, whether they subsequently proceed 
through school and graduate, the extent of child abuse and neglect in the community, and the fraction 
of children who are involved in violent crime. By summarizing these and other key aspects of 
families’ experiences, communitywide measures provide the overview planners and administrators 



! In addition to providing the funding for this paper. The Pew Charitable Trusts, along with the 
Center for Assessment and Policy Development (CAPD), helped to guide our thinking on these 
issues throughout the design phase of The Children’s Initiative evaluation. In particular, we are 
indebted to Patricia Patrizi of the Trusts and Susan Stephens of CAPD for their support and 
encouragement. The introductory quotations were first juxtaposed by Gardiner (1970). 



O 

EKLC 



1 



55 



require in order to understand the characteristics of the families in a community, develop programs 
and policies to address problems, and assess the effectiveness of these programs and policies. 

Communitywide measures provide an important perspective but, as summaries, they capture only 
a few aspects of what it is like for families to raise children in a community. Furthermore, these 
broad measures tend to mask variation among families and interactions between characteristics. For 
example, a measure may show that 20 percent of the families in a community are poor, without 
indicating the extent to which their poverty is transitory or prolonged, the degree to which they 
interact to support one another, or the variation in employment prospects among families. 

This gap between the measures and the reality they seek to describe challenges analysts to 
develop an appropriate set of measures and the means to interpret them. No single measure can 
adequately describe a community; only a broad and diverse set of measures can summarize the factors 
and characteristics that determine the well-being of residents. At the same time, this set must not 
be so large that it fails to provide an understandable indicator of community status or so large that 
the required data cannot be easily collected. The information that results from community measures 
must also be presented in a way that effectively summarizes the status of children and families in the 
community without overstating the ability of the summary measures to represent something as 
complex as a community. A balanced interpretation of this information should reflect the strengths 
and weaknesses of the specific measures and draw on information obtained through direct observation 
in the community. 

This paper provides a step toward addressing this challenge by proposing a set of measures that 
describe the general status of children and families in a community. Specifically, this set captures the 
following six aspects of child and family well-being: 

1. Child and family health 

2. Family functioning 

3. Child development 



4. School performance 

5. Youth maturation and social integration 

6. Overall socioeconomic environment 

In presenting the community measures, we begin with a review of the criteria used to select 
measures and an overview of the specific measures. In subsequent chapters, we discuss each of the 
six domains, including the specific measures, possible data sources, and measurement issues. We also 
discuss empirical evidence showing that a particular measure may be changed by policy interventions. 

Four general recommendations emerge from our review. First, the process of selecting the 
specific measures to collect must be consistent with the intended uses of those measures. When the 
measures are intended as a basis for developing policy or for understanding the current status of 
children and families in a community, then communities should use as many of the measures included 
in this report as possible. When the measures are intended to evaluate program performance and 
to represent shared community goals for system change, then only a community can make the 
ultimate choice of measures. Externally imposed measures are likely to lack sufficient standing in the 
community to serve as viable goals for evaluating a complex and potentially disorienting process like 
reforming the system of services for children and families. Thus, efforts at system reform can draw 
on the menu of measures provided in this report, but must ultimately work with the community to 
select the specific goals and measures that will be used to guide and evaluate the reform. 

Second, measurement of many aspects of child and family well-being remains unsettled. This is 
particularly true in the domains of family functioning and child development where there are many 
competing measures and few clearly established benchmarks. Work is underway to improve these 
measures and the next decade should see substantial improvements in measurement technology. In 
the meantime, we have identified a representative set of measures that have proven to be useful for 
measuring the key aspects of child and family well-being. Communities should therefore view this 
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set of measures as a work in progress and may find alternative measures that more aecuratelv capture 
their specific goals and objectives. 

Third, communities interested in comprehensive systems reform should collect information from 
all six domains. Broadly designed efforts to improve the well-being of children and families will 
require a similarly broad set of outcome measures. Such an array of measures will help communities 
ensure that reform efforts take a comprehensive view of family well-being and thereby avoid the 
narrow focus that can come from looking only at a specific domain like health of school performance. 

Fourth, communities will need to conduct surveys in order to develop a comprehensive profile 
of their children and families. While official records systems can provide data about many aspects 
of child and family well-being, many aspects of family life lie far outside the official records systems 
and can only be assessed through surveys. In addition, surveys provide a means for collecting 
information about people who are not currently participating in any formal community program or 
activity. For example, surveys would enable a community to measure the development of children 
who are not yet in school and the characteristics of families that are largely isolated from the 
community. Finally, surveys provide a means for communities to assess the extent to which 
community residents know about available services and are satisfied with those services that are 
provided. 

A. CRITERIA FOR SELECTING MEASURES 

Our primary criterion for selecting measures was that they yield useful data for communities to 
improve the lives of children and families. Specifically, we looked for measures with the following 
attributes: 

• Policy Relevance. The measure captures an aspect of children’s and families’ lives that can 
be influenced by policy or that affects the development and implementation of policy. 

• Feasibility. The measure uses data (or data collection methods) available to local 
communities. 



• Basis for Comparison. Communities can obtain data about the measure for national or 
other relevant samples which provide a frame of reference for interpreting local data on 
the measure. 

The most important criterion was that the measures be relevant for shaping policies. Community 
resources are too scarce to waste on collecting and analyzing community-level measures that arc 
irrelevant to policy formation. In searching for policy-relevant measures, we first focused on ones 
that could be used to evaluate communitywide initiatives to change the service system. 2 This process 
led us to identify measures for which there was evidence showing that program or policy interventions 
could change the measures. As we progressed, we also identified measures, such as crime rates and 
infant mortality, that are of special interest to policymakers, despite the fact that recent policy efforts 
to change these measures have been largely unsuccessful. We added these measures to our list 
because, despite their imperviousness to change, they are important aspects of child and family well- 
being and will be of continuing interest to communities. 

In assessing feasibility, we looked for communitywide measures that could be developed using 
information communities were already collecting. We also looked for measures that reflect the status 
of all children and families in a community, rather than specific subgroups, such as families receiving 
Aid to Families with Dependent Children (AFDC). We did so because of an overriding interest in 
the community as a whole, regardless of current levels of participation in various programs or 
organizations. Data for specific subpopulations of the community may misrepresent the status of all 
families and provide a poor basis for policy development. Furthermore, communities are often 
interested in the extent to which families participate in specific activities (such as how often pregnant 
woman receive adequate prenatal care). Such participation rates can only be estimated by looking 
at all eligible persons, rather than only at those using a particular service. 



2 In fact, we began to identify community-level measures of child and family well-being in order 
to evaluate The Children’s Initiative. Carcagno et al. (1993) provide an overview of the preliminary 
evaluation design, including the ways in which many of the measures reported in this paper would 
have been used to evaluate the Initiative’s efforts to improve the lives of children and families. 
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A number of data sources typically maintained by communities provide a comprehensive view 
ol children and families. These sources include birth certificates, death certificates, school registration 
forms, public health surveillance data, child abuse reports, school records, police crime reports, and 
census data. However, efforts to develop the full range of measures identified in this paper will 
require new data collection efforts, including community surveys. In particular, surveys will be 
required to measure program participation rates (for example, the fraction of eligible families using 
a family support center) and the more private aspects of family activities and characteristics (for 
example, family functioning and isolation). 

The use of surveys represents the largest change for most communities. Nevertheless, many 
communities currently use surveys to measure characteristics of their populations. For example, 43 
states and 13 large cities now conduct surveys of students in grades 9 through 12, as adjuncts to the 
national Youth Risk Behavior Surveillance System conducted by the Centers for Disease Control and 
Prevention (CDC). Communities may also use specific surveys to assess community needs or to 
measure changes in community outcomes following initiatives to improve child and family well-being. 

In identifying possible survey measures, we sought to select measures that are valid, reliable, 
practical to administer (typically this meant that questions could be asked over the telephone), not 
too time-consuming, culturally and linguistically appropriate for a wide variety of children and 
families, and developed with an appropriate normative group. We also looked for measures that have 
been applied in large-scale data collections (to demonstrate usefulness and provide comparison data), 
used in evaluation research, and shown to detect the effects of program interventions on children and' 
families. 3 

Finally, we looked for measures that communities could augment with interpretive information. 
By themselves, many of the measures are difficult to understand without some frame of reference. 
Over time, as a community develops a series of annual estimates, the observed trends will provide this 

3 

Appendix A describes how to obtain additional information about the various surveys mentioned 
in this report. 



frame of reference. In addition, national estimates can provide a useful reference point, particularly 
for general trends. For example, a community trying to interpret a rising level of youth violence 
should consider previous local levels of violence and changes at the national level. 

In addition to the three criteria for selecting specific measures, we considered the overall 
rationale communities would use to justify the considerable effort required to collect outcome 
measures. Communities must balance the value of the information gained from collecting these 
measures with the costs of doing so. The costs are often obvious, consisting of staff time and other 
resources required to assemble data from existing sources or to conduct a community survey. 
Benefits, although less obvious, can also be substantial. In particular, the measures can provide 
communities with a basis for targeting their resources and monitoring the effectiveness of their 
programs. Failure to collect outcome data may result in missed opportunities to help or, even worse, 
in misdirected or squandered resources. Furthermore, clear evidence of improving outcomes pr 
program success is essential to building support for policies, just as evidence of their failure weakens 
public support. 

The growing demand for community-level measures suggests that, in many cases, communities 
feel that the benefits outweigh the costs. Clearly, efforts to institute results-based accountability 
require community-level measures that can serve as specific goals for programs seeking to improve 
the lives of children and families (Schorr 1994). Also, the many communities that are initiating new 
efforts to improve their systems for addressing the needs of children and families will require 
community measures to target and evaluate those initiatives. In late 1993, the Families and Work 
Institute (1993) identified 8 national initiatives and 43 state and local initiatives to improve outcomes 
for children and families. State and local governments also need community-level measures about 
children and families in order to set priorities, target resources, and evaluate programs. Schorr (1994) 
and her colleagues at the Harvard Project on Effective Services mention several specific benefits of 
collecting information on community-level outcomes: 



• Measurable outcomes provide a foundation for a system of results-based accountability, 
which in turn can replace or diminish the need for centralized bureaucratic 
micromanagement. 

• Community outcome measures (particularly time trends) can assure funders and the 
public that social investments are producing the desired results. 

• Well-defined outcome measures help foster agreement among community agencies on 
cross-system collaboration. 

• Outcome measures can help minimize efforts that do not contribute to improved results, 
by directing attention away from activities that do not affect the measures and toward 
activities that do. 

• Outcomes play a key role in evaluations of program effectiveness. 

• Outcome measures help to avoid inappropriate expectations, by providing clear program 
goals that can be used to assess whether a program’s resources will be sufficient to meet 
these goals. 

In the end, the final decision to collect community-level outcome measures must rest with the 
community. We have proposed a list of feasible measures, but communities must select the ones that 
best represent the goals they seek to achieve. Communities can use the measures listed here to 
provide a framework for deciding on community goals and targets. However, the measures will lack 
sufficient force as goals for system change or program evaluation if they are not selected to represent 
the interests of the community. In making any decisions about specific measures, however, 
communities must be mindful of the need for comprehensiveness. While communities may decide 
not to pursue all the measures in each of the domains, it is essential to pursue at least some measures 
from each domain. Child and family well-being is multidimensional and only by producing a 
comprehensive set of measures will the community be able to judge its progress toward improving the 
lives of children and families. 

B. OVERVIEW OF THE SELECTED MEASURES 

Overall, we identified more than 60 community-level measures (Table 1.1). Measures that can 
be obtained from the records systems typically maintained by communities are noted with an asterisk 
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in the table. The following chapters provide additional information about the measures and evidence 
of their susceptibility to policy interventions. 

Of course, the selection of a specific set of outcome measures is only the first step in using them. 
Communities must decide what to make of changes in the measures over time and how to use the 
measures to evaluate success. This is a difficult challenge because of the dynamic nature of 
communities and the many factors that can influence the outcomes listed in Table 1.1. In particular, 
it is often impossible to know how much a change in outcomes reflects the effect of a specific 
community intervention, the effect of another intervention or a change in community characteristics, 
or more-or-less random fluctuation in the outcome measure over time. As Hollister and Hill (1994) 
point out, there are no evaluation methods currently available that will estimate community-level 
impacts with a known degree of statistical precision. As a result, communities will be left with 
substantial uncertainty in interpreting patterns of outcome measures, particularly if they want to tie 
outcome changes to specific policy actions. 

Despite the uncertainty in any impact estimates, the outcome measures provide an index of the 
well-being of children and families that is likely to be useful to communities. As noted in the 
introduction to this paper, such measures provide a means for understanding the community and for 
setting priorities. Furthermore, communities can use several approaches to develop a general sense 
of how well current programs (or a new initiative) are doing. These approaches include: 

• Assess local trends in outcome measures to determine whether changes are in the desired 
direction (for example, whether high school graduation rates are rising over time). 

• Compare local trends with trends in the nation or other communities (for example, 
whether the rate of low birthweight in a community is decreasing faster than the change 
observed for the nation as a whole or for other similar communities). 

• Compare measures for the community with specific absolute standards (for example, 
whether there are any cases of measles or whether 90 percent of pregnant women 
receive adequate prenatal care, as specified in the nation’s health care goals, Healthy 
People 2000). 
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• Look at intermediate variables, such as service-use patterns, that may predict long-term 
changes in key outcome variables (for example, information about immunization rates 
can be collected in the short term and should predict longer-term changes in the 
incidence of certain communicable diseases). 

• Set realistic goals that recognize it takes time to affect community conditions that have 
come about over a long time. 

None of these strategies offers a precise method for interpreting changes in outcome measures. 
In all cases, analysts will have some level of uncertainty about what a measure means and how to 
interpret change in that measure over time. However, these strategies provide a basis for 
systematically considering the measures and thinking about what they mean for a community. 

It is useful to note that our list excludes many important community-level measures. Specifically, 
we have omitted many measures that pertain primarily to adults, including adult crime rates, overall 
patterns of substance abuse, and local economic conditions. The list also excludes the types of 
detailed information collected by direct observation of community activities. For example, it does not 
include observations about the extent to which health care providers accept Medicaid, the ease of 
access to key community institutions, and the degree of cooperation between levels of government 
and community residents. These other measures provide a context for interpreting the measures in 
our list that pertain to children and families. A full interpretation of the outcome measures listed 
in Table 1.1 will require a sense of these other community characteristics. 
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TABLE VIM 



A COMMUNITY'S SOCIOECONOMIC ENVIRONMENT MEASURING 
DEMOGRAPHIC TRENDS AT LOCAL LEVELS 



Level of 



Characteristics 


National Estimate 
(Year) 


Published Data 
Available 


Source 


Population 








Number of Persons 


255 million (1992) 


C,N a 


Census of Population & Housing 


Number of Family Households 


65 million (1990) 


C,N 


Census of Population & Housing 


Persons Who Have Moved (Percentage) 6 


46.7% (1990) 


C,N 


Census of Population & Housing 


Age (Percentage of Total Population) 
Children under age 5 
Children under age 18 
Persons age 65 and over 


7.4% (1990) 

25.6% (1990) 

12.6% (1990) 


C,N 

C,N 

C,N 


Census of Population & Housing 
Census of Population & Housing 
Census of Population & Housing 


Race/Ethnicity (Percentage) 
Whites 
Blacks 

Asian and Pacific Islander 
Hispanics c 


80.3% (1990) 

12.1% (1990) 

2.9% (1990) 

9.0% (1990) 


C,N 

C,N 

C,N 

C,N 


Census of Population & Housing 
Census of Population & Housing 
Census of Population & Housing 
Census of Population & Housing 


Gender 








Males as a percentage of females 


95.1% (1990) 


C,N 


Census of Population & Housing 


language (Percentage) 

Households speaking a language at home 
other than English 


13.8% (1990) 


C,N 


Census of Population & Housing 


Income 








Per Capita Income 


$ 14,420 (1989) 


C,N 


Census of Population & Housing 


Median Family Income 


$ 35,225 (1989) 


C,N 


Census of Population & Housing 


Percentage with Income Below the Poverty 
Level 


13.1% (1989) 


C,N 


Census of Population & Housing 


Education (Percentage) 








High School Graduate or Higher 


75.2% (1990) 


C,N 


Census of Population & Housing 


Bachelor’s Degree or Higher 


20.3% (1990) 


C,N 


Census of Population & Housing 


Crime 








Serious Crimes Reported per 100,000 Persons 


5928 (1991) 


C 


County and City Data Book d 


Employment 








Unemployment Rate' 


6.7% (1991) 


C,N 


Census of Population & Housing 


Manufacturing Jobs (Percentage of Labor 
Force) 


17.7% (1990) 


C,N 


Census of Population & Housing 


Labor Force (Percent Change from 1980) 


18.0% (1990) 


C,N 


Census of Population & Housing 



TABLE VII. 1 (continued > 



Characteristics 


National Estimate 
(Year) 


Level of 
Published Data 
Available 


Source 


Infrastructure 

I xxral Government Direct General Expenses f 


$392 billion (1986-87) 


C 


County* and City Data Book p 


Housing 

Number of units 


$102 million (1990) 


C,N 


Census of Population <£ Housing 


Owner-occupied units (percentage of total 
occupied units) 


64.2% (1990) 


C,N 


Census of Population & Housing 


Hospital Beds per 100,000 Persons 


366 (1991) 


C 


County and City Data Book h 


Active Physicians per 100,000 Persons 


214 (1990) 


C 


County and City Data Book ' 



Note: The sources provide nationally compiled data that are readily available and easily accessible for a community. Each state. 

however, compiles its own data and may provide more detailed types of information at the zip code or neighborhood level. 
Variables from the Census of Population and Housing are accessible in CD-ROM format and can also be found in the County 
and City Data Book. Most of the variables are compiled and updated approximately every five years. 

3 A ‘C* signifies that the data are available at the county, city, and metropolitan area level, and an ‘N’ signifies that the data are available 
at the neighborhood level and the zip code level. The neighborhood level, unless otherwise noted, refers to a census tract or a group of 
census tracts that have been aggregated to form a subcity area. 

b Measured in 1990, these persons were living in a different home in 1990 than in 1985. 
c Persons of Hispanic origin may be of any race. 

d The County and City Data Book compiles unpublished data from the U.S. Federal Bureau of Investigation’s Uniform Crime Reporting 
Program. 1 6 

c Civilian unemployed are reported as a percentage of total civilian labor force. 

f More detailed data on local government finances are available in the Census of Governments , generally published in years ending in 2 
and 7. 

*The County and City Data Book compiles data from the Census of Governments, Government Finances , Compendium of Government 
Finances , conducted every fifth year (for years ending in 2 or 7). 

These data were originally published by the American Hospital Association (AHA). More recent figures, available on an annual basis, 
can be calculated from published AHA data. See the American Hospital Association’s Hospital Statistics. 

1 These data were originally published by the American Medical Association (AMA). More recent figures, available on an annual basis, 
can be calculated from published AMA data. See the AMA's Physician Characteristics and Distribution in the United States. 
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